If you're looking for a reason the world will suddenly end, it's not hard to find one--especially if your job is to convince people they need to buy things to prepare for the apocalypse. "World War III, China, Russia, Iran, North Korea, Joe Biden--you know, everything that's messed up in the world," Ron Hubbard, the CEO of Atlas Survival Shelters, told me. His Texas-based company sells bunkers with bulletproof doors and concrete walls to people willing to shell out several thousand--and up to millions--of dollars for peace of mind about potential catastrophic events. Lately, interest in his underground bunkers has been booming. "When the war broke out in Ukraine, my phone was ringing every 45 seconds for about two weeks," he said.

I expect navigating the acute risk period to be tricky for our civilization, for a number of reasons. Success looks to me to require clearing a variety of technical, sociopolitical, and moral hurdles, and while in principle sufficient mastery of solutions to the technical problems might substitute for solutions to the sociopolitical and other problems, it nevertheless looks to me like we need a lot of things to go right. For instance, people are still regularly surprised when I tell them that I think the hard bits are much more technical than moral: it looks to me like figuring out how to aim an AGI at all is harder than figuring out where to aim it.[1] Within the list of technical obstacles, there are some that strike me as more central than others, like "figure out how to aim optimization". And a big reason why I'm currently fairly pessimistic about humanity's odds is that it seems to me like almost nobody is focusing on the technical challenges that seem most central and unavoidable to me.

WARNING: Reading this article may commit you to an eternity of suffering and torment. These are some of the urban legends spawned by the Internet. Yet none is as all-powerful and threatening as Roko's Basilisk. For Roko's Basilisk is an evil, godlike form of artificial intelligence, so dangerous that if you see it, or even think about it too hard, you will spend the rest of eternity screaming in its torture chamber. Even death is no escape, for if you die, Roko's Basilisk will resurrect you and begin the torture again.

We at MIRI are soliciting help with an AI-alignment project centered around building a dataset, described below. We have $200,000 in prizes for building the first fragments of the dataset, plus an additional $1M prize/budget for anyone who demonstrates the ability to build a larger dataset at scale. If this project goes well, then it may be the first of a series of prizes we offer for various projects. Below, I'll say more about the project, and about the payouts and interim support we're offering. Hypothesis: Language models can be made more understandable (and perhaps also more capable, though this is not the goal) by training them to produce visible thoughts.

If you've ever used Skype or shared files on Kazaa back in the early '00s, you've encountered the work of Jaan Tallinn. And if humans wind up creating machines that surpass our own intelligence, and we live to tell about it -- we might have Tallinn's philanthropy, in small part, to thank. Tallinn, whose innovations earned him tens of millions of dollars, was one of the first donors to take seriously arguments that advanced artificial intelligence poses a threat to human existence. He has come to believe we might be entering the first era in human history where we are not the dominant force on the planet, and that as we hand off our future to advanced AI, we should be damned sure its morality is aligned with our own. He has donated more than $600,000 to the Machine Intelligence Research Institute, a prominent organization working on "AI alignment" (that is, aligning the interests of an AI with the interests of human society) and more than $310,000 to the Future of Humanity Institute at Oxford, which works on similar subjects.

Traditional models of rational action treat the agent as though it is cleanly separated from its environment, and can act on that environment from the outside. Such agents have a known functional relationship with their environment, can model their environment in every detail, and do not need to reason about themselves or their internal parts. We provide an informal survey of obstacles to formalizing good reasoning for agents embedded in their environment. Such agents must optimize an environment that is not of type ``function''; they must rely on models that fit within the modeled environment; and they must reason about themselves as just another physical system, made of parts that can be modified and that can work at cross purposes.

Stephen Hawking has said, "The development of full artificial intelligence could spell the end of the human race." Elon Musk claims that AI is humanity's "biggest existential threat." That might have people asking: Wait, what? But these grand worries are rooted in research. Along with Hawking and Musk, prominent figures at Oxford and UC Berkeley and many of the researchers working in AI today believe that advanced AI systems, if deployed carelessly, could end all life on earth. This concern has been raised since the dawn of computing. But it has come into particular focus in recent years, as advances in machine-learning techniques have given us a more concrete understanding of what we can do with AI, what AI can do for (and to) us, and how much we still don't know. Some of them think advanced AI is so distant that there's no point in thinking about it now. Others are worried that excessive hype about the power of their field might kill it prematurely. And even among the people who broadly agree that AI poses unique dangers, there are varying takes on what steps make the most sense today.

Algorithmic Justice League is a collective started that aims to remove human bias from AI algorithms that can result in exclusionary experiences and discriminatory practices. It focuses on 3 key areas 1) Highlight Algorithmic Bias through Media, Art, and Science 2) Provide Space for People to Voice Concerns and Experiences with Coded Bias, 3) Develop Practices for Accountability During the Design, Development, and Deployment of Coded Systems. AI Now Institute at New York University is an interdisciplinary research center dedicated to understanding the social implications of artificial intelligence. Their work focuses on four core domains: Rights & Liberties, Labor & Automation, Bias & Inclusion, Safety & Critical Infrastructure. AI Ethics Lab brings together researchers and practitioners from various disciplines to detect and solve issues related to ethical design in AI. Based in US and Turkey, the Lab offers a comprehensive approach to ethical design of AI-related technology.

When Nate Soares psychoanalyzes himself, he sounds less Freudian than Spockian. As a boy, he'd see people acting in ways he never would "unless I was acting maliciously," the former Google software engineer, who now heads the non-profit Machine Intelligence Research Institute, reflected in a blog post last year. "I would automatically, on a gut level, assume that the other person must be malicious." It's a habit anyone who's read or heard David Foster Wallace's "This is Water" speech will recognize. Later Soares realized this folly when his "models of other people" became "sufficiently diverse"--which isn't to say they're foolproof, he wrote in the same post.

At the EA Global 2016 conference, I gave a talk on "Using Machine Learning to Address AI Risk": It is plausible that future artificial general intelligence systems will share many qualities in common with present-day machine learning systems. If so, how could we ensure that these systems robustly act as intended? We discuss the technical agenda for a new project at MIRI focused on this question. The talk serves as a quick survey (for a general audience) of the kinds of technical problems we're working on under the "Alignment for Advanced ML Systems" research agenda. Included below is a version of the talk in blog post form.1 Actions are hard to evaluate 2.2. This talk is about a new research agenda aimed at using machine learning to make AI systems safe even at very high capability levels. I'll begin by summarizing the goal of the research agenda, and then go into more depth on six problem classes we're focusing on. The goal statement for this technical agenda is that we want to know how to train a smarter-than-human AI system to perform one or more large-scale, useful tasks in the world.